The Measurement Manager: Modular and Efficient End-to-end Measurement Services

Title of Dissertation: THE MEASUREMENT MANAGER: MODULAR AND EFFICIENT END-TO-END MEASUREMENT SERVICES Pavlos Papageorgiou, Doctor of Philosophy, 2008 Directed by: Associate Professor Michael Hicks Department of Computer Science Department of Electrical and Computer Engineering End-to-end network measurement is used to improve the precision, efficiency, and fairness for a variety of Internet protocols and applications. Whether streaming media files, constructing an overlay or picking a candidate server to download from, applications need to provide a good user experience. What constitutes a good user experience differs for each application but what is common is the need to discover and adapt to the current conditions of the network path. This is especially important since, due to the design of the Internet, network routers do not provide any feedback about these network properties. Measurement is typically performed in one of three ways: (1) actively, by injecting specially crafted probe packets into the network, (2) passively, by observing existing data traffic, and (3) customized, where applications use use their own traffic to perform customized measurements. All current approaches suffer from drawbacks. Passive techniques are efficient but are constrained by the shape of the existing traffic, limiting the speed and accuracy of their measurements. Active techniques are faster, more accurate and more flexible but impose a significantly higher overhead by competing with applications for bandwidth. And finally, custom techniques combine flexibility with efficiency, but are so tightly coupled with each application that they are not reusable. To address these shortcomings, we present the Measurement Manager, a practical, modular, and efficient service for performing end-to-end network measurements between hosts. Our architecture introduces a new hybrid approach to network measurement, where applications can pool together their data packets to be reused as padding inside network probes in a transparent and systematic way. We achieve this through the Measurement Manager Protocol (MGRP), a new transport protocol for sending probes that combines data packets and probes on the fly. In MGRP, active measurement algorithms specify the probes they wish to send using a Probe API and applications allow MGRP to use data from their own packets to fill the otherwise wasted probe padding. The ability of MGRP to piggyback any data packet on any probe is pivotal in making our measurement system unique in the sense that any measurement algorithm can now be written as if active, but implemented as if passive. We have implemented the Measurement Manager inside the Linux kernel and have adapted existing applications and active measurement tools to use our system. Through experimentation we provide detailed empirical evidence that piggybacking data packets on measurement probes is not only feasible but improves source and cross traffic as well as the performance of measurement algorithms while not affecting their accuracy. We show that the Measurement Manager is an architecture with broad applications that can be used to build a generic measurement overlay network as well as expanding the solution space for estimation algorithms, since every application packet can now act as a potential probe. THE MEASUREMENT MANGER: MODULAR AND EFFICIENT END-TO-END MEASUREMENT SERVICES